Smart-court system and method for providing real-time debriefing and training services of sport games

ABSTRACT

A Smart-court system, adaptive to constrained sport environment, for enabling real time analysis and debriefing of sport activities is provided herein. The Smart-court system is comprised of: (i) an automatic recording system comprising a plurality of video cameras located in a court, arranged to real-time (RT) recording of a sport session and utilizing automatic calibration and stabilization module; and (ii) a data processing system comprising: (a) a capture module for grabbing a video stream; (b) an objects&#39; detector module arranged to extract during the RT sport session, the objects from the foreground of each frame; (c) an event module for automatically analyzing, the motion and the activities of the tracked objects for automatically identifying and classifying events, creating a synchronized event log and calculating statistics that occurred during the RT sport session; and (d) a presentation module enabling to perform instant debriefing, combined biomechanical and tactical analysis of the video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/981,127, filed May 16, 2018, which is a continuation of U.S. patentapplication Ser. No. 14/370,006, filed Jun. 30, 2014, now U.S. Pat. No.9,999,825, which is the U.S. national phase application of InternationalApplication No. PCT/IL2013/050162, filed Feb. 21, 2013, which designatedthe United States and claims the benefit of priority to U.S. ProvisionalApplication No. 61/602,360 filed Feb. 23, 2012, the entire contents ofeach of which are incorporated herein for all purposes.

BACKGROUND Technical Field

The present invention relates generally to analysis of motion andactivities of an object that is involved in a real sport game. Moreparticularly, the present invention relates to classification andanalysis of a real sport game session.

Discussion of Related Art

Participants of real sport games as well as coaches are in an ongoingeffort to improve the activity and capability of the participants in thesport games. For that purpose, various analysis systems exist in theart.

Some of the analysis systems that exist in the market require fromparticipants of the real sport games to be equipped with an electronicsensing device. Also, some of the analysis systems use a large array ofcameras, which require a dedicated operator and a long setup andcalibration process. Because of these requirements these analysissystems are considered to be very complicated and expensive.

Furthermore, video analysis systems that currently exist in the marketrequire long preparation period before the video analysis can takeplace. For example, such preparations process may include manualrecording, calibration procedure and uploading of footage. Also, thesesystems require a user to invest time in watching and analyzing uneditedvideos and manual tagging of various events that took place during thetraining or match sessions, since the events that are of interest toparticipants and coaches are not grouped together.

BRIEF SUMMARY

According to some embodiments of the present invention, a Smart-courtsystem, adaptive to constrained sport environment, for enabling realtime analysis and debriefing of sport activities is provided herein. TheSmart-court system is comprised of: (i) an automatic recording systemcomprising a plurality of video cameras that are located in a court,arranged to real-time (RT) recording of a sport session and utilizing anautomatic calibration and stabilization module requiring no humanintervention; and (ii) a data processing system that is comprised of:(a) a capture module for grabbing a video stream of the RT sportsession; (b) an objects' detector module that is arranged to receive astream of video of the tracked objects and extract, during the RT sportsession, the objects from the foreground of each frame; (c) an eventmodule for automatically analyzing, during the RT sport game, the motionand the activities of the tracked objects that appear in the stream ofmultimedia data for automatically identifying and classifying eventsbased on the tracking and analysis of the motion and the activities ofthe tracked objects, creating a synchronized event log and calculatingstatistics that occurred during the RT sport session; and (d) apresentation module enabling to perform instant debriefing, combinedbiomechanical and tactical analysis of the video by selectively playingspecific events by type based on the synchronized event log.

According to some embodiments of the present invention, the smart courtsystem is further comprising a tracking module for identifying in realtime basic events enabling to identify a game session starting with astroke and ending when one of the players gained a score, wherein theevent module enables to identify complex events in near real time at theend of each session during the game match.

According to some embodiments of the present invention, the online videoassociated with event log is uploaded and published in one or moresocial networks.

According to some embodiments of the present invention, the event moduleis further arranged to yield a pattern of behavior and a tacticalbehavior of a player from more than one event.

According to some embodiments of the present invention, the dataprocessing system is further generating a profile of the playerincluding information related to: the player's tactical behavior, thepattern of behavior, game statistics, calculated success rate incomparison with other players and events.

According to some embodiments of the present invention, cameracalibration is enabled by dividing a court into grids and checking eachjunction of in the grid for a deviation and wherein the deviation isidentified in the grid than the frame in the streamed video may bechecked in the streamed video.

According to some embodiments of the present invention, the dataprocessing system identify start and end of a real sport game or realtraining and automatically identify change of status of the real sportgame.

According to some embodiments of the present invention, the video thatis transmitted from the cameras is automatically customized to adjust toenvironment conditions and does not require an intervention of anoperator.

According to some embodiments of the present invention, vibration in thestream of video is automatically reduced.

According to some embodiments of the present invention, the dataprocessing module further enables to create automatic customized videomovie by integrating event or segments according to given criteria whichare determined by the user or predefined by the system.

According to some embodiments of the present invention, change ofdesignated location of players is identified when the players switchsides on the court.

According to some embodiments of the present invention, an additionaltechniques and strategies module provides recommendations to a playerhow to correct activity in a real sport game based on events' analysis.

According to some embodiments of the present invention, events' analysisand pattern behavior enable cutting out idle time in the video file tosave storage space.

According to some embodiments of the present invention, the presentationmodule enable playing abridged video which displays only selectedspecified events by jumping between the events according to user querybased on the synchronized log event.

According to some embodiments of the present invention, the presentationmodule enables playing a virtual 3D dynamic scenes using a 3D enginewhich simulates objects behavior based on analyzed video data of objectsmotion and the synchronized event log.

According to some embodiments of the present invention, the patternbehavior of the player is further analyzed to identify mental events.

According to some embodiments of the present invention, a method forenabling real time analysis and debriefing of sport activities in aSmart-court system that is adaptive to constrained sport environment isprovided herein. The method is comprised of the following stages: (i)automatically recording in real time (RT) a real sport session byutilizing an automatic calibration and stabilization module requiring nohuman intervention by a system that is comprising a plurality of videocameras that are located in a court; (ii) tracking motion and theactivities of objects in a video of the RT sport session by a trackingmodule; (iii) receiving a stream of video of the tracked objects andsimultaneously analyzing, during the RT sport session, the motion andthe activities of the tracked objects that appear in the stream ofvideo; (iv) automatically identifying and classifying events based onthe tracking and analysis of the motion and the activities of thetracked objects and calculating statistics that occurred during the RTsport session; and (v) creating a synchronized event log of theclassified events, wherein a user is enabled to perform instantdebriefing, combined biomechanical and tactical analysis of the video byselectively playing specific events by type based on the synchronizedevent log.

According to some embodiments of the present invention, basic events areidentified in real time, for identifying game session starting with astroke and ending when one of the players gained a score, whereincomplex events are identified near real time at end of each sessionduring the game match.

According to some embodiments of the present invention, the method isfurther playing an abridged video by jumping between different eventsbased on a user query according to the synchronized event log.

According to some embodiments of the present invention, the method isfurther including publishing and uploading, game statistics, the onlinevideo and associated event log in one or more social networks.

According to some embodiments of the present invention, the method isfurther yielding a pattern of behavior and a tactical behavior of aplayer from more than one event by the analysis module.

According to some embodiments of the present invention, the method isfurther generating and uploading to the network a profile of the playerand saving information related to: the player's tactical behavior, thepattern of behavior, performance statistics and events by the dataprocessing system.

According to some embodiments of the present invention, the method isfurther identifying start and end of a real sport game or real trainingand automatically identify change of status of the real sport game bythe data processing system.

According to some embodiments of the present invention, the method isfurther automatically customizing the video that is transmitted from thecameras to adjust to environment conditions and does not require anintervention of an operator.

According to some embodiments of the present invention, the method isfurther automatically reducing vibration in the stream of video.

According to some embodiments of the present invention, the method isfurther identifying the change of status by a predefined gesture of theuser.

According to some embodiments of the present invention, the method isfurther identifying change of designated location of players, when theplayers switch sides on the court.

According to some embodiments of the present invention, the method isfurther providing recommendations to a player how to correct activity ina real sport game based on events' analysis by an additional techniquesand strategies module.

According to some embodiments of the present invention, the method isfurther cutting out idle time in the video file to save storage space byevents' analysis and pattern behavior.

According to some embodiments of the present invention, the method isfurther playing an abridged video which includes specified events.

According to some embodiments of the present invention, the method isfurther playing a virtual 3D dynamic scenes based on the videos using 3Dengine to simulate objects behavior based analyzed motion data ofobjects and the synchronized event log.

According to some embodiments of the present invention, the method isfurther analyzing pattern behavior and tactical behavior to identify amental event of the player.

According to some embodiments of the present invention, the method isfurther comprising the step of creating customized video movie byintegrating event or segments according to given criteria which aredetermined by the user or predefined by the system.

According to some embodiments of the present invention, the method isfurther comprising the step of identifying unforced error based onidentifying the timing and position of the player before striking theball.

According to some embodiments of the present invention, the method isfurther comprising the step of real time automatic calculation of gamescore based on identified classified events and analysis of player andball position and motion.

According to some embodiments of the present invention, the method isfurther comprising the step of comparing between the players' profilesand providing performance matching between players.

These, additional, and/or other aspects and/or advantages of the presentinvention are: set forth in the detailed description which follows;possibly inferable from the detailed description; and/or learnable bypractice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more readily understood from the detaileddescription of embodiments thereof made in conjunction with theaccompanying drawings of which:

FIG. 1A is a top 3D perspective diagram of a system incorporated in aTennis court, according to some embodiments of the invention;

FIG. 1B is a block diagram of a smart-court system, according to someembodiments of the invention;

FIG. 2 is a flowchart diagram of objects detecting module, according tosome embodiments of the invention;

FIG. 3 is a flowchart diagram illustrating activity of game trackermodule, according to some embodiments of the invention;

FIG. 4 is a flowchart diagram illustrating activity of event module,according to some embodiments of the invention;

FIG. 5 is a flowchart diagram illustrating activity of moments module,according to some embodiments of the invention;

FIG. 6 is a flowchart diagram illustrating activity of upload module,according to some embodiments of the invention; and

FIG. 7 is a flowchart diagram illustrating activity of Debriefingpresentation training module, according to some embodiments of theinvention.

DETAILED DESCRIPTION

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of construction and the arrangement of the components setforth in the following description or illustrated in the drawings. Theinvention is applicable to other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for the purpose ofdescription and should not be regarded as limiting.

The term “event” as used herein in this application is defined as a atleast one action which occur during a real sport game as recorded in amultimedia file. The beginning and the end of the event is updated inevent log file. Also, various parameters which are related to the eventare saved in the log event file.

The term “classification of event” as used herein in this application isdefined as an attribution of an event to a predefined type of sequenceof actions that occur in a real sport game. The classification of theevent is a result of an analysis of the event and it is saved in thedatabase.

The term “tactical behavior” as used herein in this application isdefined as a result of analysis of a pattern of behavior of a playeraccording to more than one event. The pattern of behavior may beclassified according to a predefined list of tactical behavior types andthe classification of the tactical behavior is saved in the databasealong with the players' profile and other details of the real sportgame.

The term “profile” as used herein in this application is defined as acollection of parameters which are related to a real player in a realsport game.

The term “T Point” as used herein in this application is defined as thespot on a tennis court where the center line and the service lineintersect perpendicularly to form a “T” shape.

The term “session” as used herein in this application is defined as asegment in a real sport game which starts in a strike and ends in earnedpoints.

The term “near real-time” (NRT), as used herein in this application isdefined as a delay that is introduced by automated data processingbetween the occurrence of an event and the use of the processed data.For example, classifying an event as a near real-time event refers tothe real time of event occurrence, minus the processing time, as nearlythe time of the live event.

According to some embodiments of the present invention, a system forenabling simultaneous analysis and debriefing of sport activities isprovided herein. The system is enabling a user to track all events thatoccurred during an entire match (i.e. real sport game) or training.Furthermore, the system is providing automatic classification, analysisand understanding the events that occurred during a real sport game(i.e. a match) or training, for improving the performance of the playerin the real sport game.

According to some embodiments of the present invention, the system mayalso record and analyze activity and motion of a tennis player and otherobjects which are related to the real sport game, by utilizing streameddata.

According to some embodiments of the present invention, there isprovided a data processing system. The data processing system may bearranged to receive a stream of video of a real time (RT) sport sessionand to simultaneously analyze, during the sport session, motion andactions of objects that appear in the streamed multimedia file.

According to some embodiments of the present invention, the system isimplemented by utilizing a smart-court platform that is adaptive to aconstrained sport environment. The constrained sport environment isincluding at least two cameras, preferably four cameras, for recordingthe real sport game. A constrained sport environment may be a swimmingpool, a football field, a tennis court etc. For the purpose ofexplaining the present invention, we will refer in a non limitingexample to a tennis game or tennis training as a real sport game.

According to some embodiments of the invention, vision and imageprocessing algorithms are applied for analysis of events that occurredin the real sport game.

Embodiments of the present invention provide a smart-court platform foranalyzing patterns of behavior of an object that is participating in areal sport game and classifying the patterns. The smart-court platformis receiving a multimedia stream of the sport game from cameras whichare located in the place where the real sport game is taking place. Thesystem may automatically record and analyze a pattern of tacticalbehavior and then classify it.

According to some embodiments of the present invention, the systemprovides calculations of game statistics such as: (i) player's andball's position in various positions; (ii) player's speed; (iii)player's mileage calculation; (iv) player's acceleration; (v) ball'slocation and spin; (vi) number of winners; (vii) number of double fault;and (viii) serve percentage, speed, position and points on first servei.e. whether the player won the point after the serve.

The game statistics may be calculated in real time and may be updated inreal time throughout the sport game.

According to some embodiments of the present invention, the systemautomatically classifies events that took place during an entire matchor training i.e. session, including: player behavior related events orgame rules, session events including: (a) types and order of strokes,(b) net approach; (c) passing shot; (d) slice; and (e) aces, (f)successfully gained points; (g) game type; (h) match; (i) tacticalpatterns; (g) forced error; and (h) unforced error. An unforced error isa description of poor performance of a player and may be identified inrelation to a player's profile.

The unforced error may be detected by analyzing the player's positionand timing before striking the ball, in case the player has reached thepoint and position where the player can strike the ball, at leastspecified amount of time (e.g. one second) before the ball and fails inher or his performance, an unforced error is identified. In case theplayer reached the striking point less than said specified amount oftime (E.g. one second) and fails, a forced error is identified. Thisalgorithm of identifying forced and unforced error is based on analysisof an accurate identification of the positions and motion of objects(i.e. players and balls) at all frames. The system may provide adetailed analysis of: (i) all the serves that hit the T point, Widepoint or body point (each serve can be close to serve line or to thenet); and (ii) all the serves above certain speed. Type of strokes mayinclude: (i) first and second serve; (ii) backhand; (iii) forehand,including: inside out and passing shots; (iv) forehand and backhandvolley; (v) nets; (vi) first and second return; (vii) smash.

According to some embodiments of the present invention, the system mayfurther provide an analysis by pattern of: (i) different combo; (ii)rally; (iii) data of all points that were gained at second serves; (iv)a tactical pattern e.g. “Serve—Return—Forehand”; (v) player's position(The position may be presented in a color map) (vi) serve and run; and(vii) all rallies above a certain amount of strokes.

According to some embodiments of the present invention, the system mayfurther provide an analysis by score. For example, (i) keep score (ii)analysis according to specific score in a game; (iii) ending score in asession; (iv) big points; (v) outs and nets; and (vi) leading and behindstates.

According to some embodiments of the present invention, the analysis maybe performed for identifying and classifying events that occurred duringan entire session. Further, the analysis may result in an online editedvideo version of the RT sport session. Additionally, a user may beenabled to perform debriefing and analysis operation by selecting toreview specific events by their type according to the event log.

According to some embodiments of the present invention, an access to allevents of the same type may be achieved via: (i) a clip with a sequenceof all events from the same type, or (ii) an abridged video whichutilizes events' log file, where a user may “jump” between all eventswhich are from the same type. Each event in the events log file mayinclude indication of a start point of an event and an end point of theevent. Skipping between events may be enabled to the user based onautomatic classification and tagging of beginning and end of each event.Each event may be associated to data including a type name according toclassification of the event, time tag, one or more players that areassociated to the event and event result.

According to some embodiments of the present invention, there isprovided an immediate feedback process which does not require pre orpost game manual processing for breaking of a session into classifiedevents. The automatic event classification process allows a player and acoach to watch the events and debrief only the events of interest.Additionally, such feedback provides an efficient and focused trainingprocess.

According to some embodiments of the present invention, the debriefingis performed via a visualization and feedback dashboard which providethe user with following information: game's statistics, events'analysis, game' video, events' log and a 3D animated virtual dynamicscenes which show three dimensional (3D) graphics display, based on theplayer's location and stroke classification, including ball's movement.

According to some embodiments of the present invention, an analysis ofthe entire match (i.e. sport game), the strategy and key points isprovided herein. The analysis is also performed to present a combinedbiomechanical and 3D tactical analysis, in a synchronized video and 3Dvirtual display. Biomechanics aspects of the match include the movementanalysis of the player for improving the player's action behavior suchas hand movement when preforming a strokes etc.

According to some embodiments of the present invention, data collectionmay be achieved in a non-intrusive manner. It does not interfere withthe game's action and it does not require the player wearing a sensorduring a match (i.e. real sport game) or training. In the 3D virtualdynamic scenes the player and ball position, the players' footwork, theball's trail and hit point on the ground are displayed. The 3D virtualdynamic scenes may be presented and displayed various viewing modes: topview, side view, from the ball's point of view and from the player'spoint of view. The 3D virtual dynamic scenes may be displayed from thepoint of view of the opponent player.

Optionally, the system may create an attributed profile of the object,based on at least one pattern of tactical behavior. The attributedprofile may include performance data such as speed of ball strokes,results' statistics, and motion pattern across the court etc. The systemmay enable a real-time upload of (i) the stored position; and (ii) theidentified motion.

Utilizing cloud computing services or any other technology, the datathat was uploaded may be accessible from any place via the Internet onany type of computer device such as laptops, tablets, smartphones andthe like. Players and coaches will be able to conduct remote debriefingand to provide remote guidance in RT. In other words, virtual coachingmay be provided.

Additionally, the system may perform a combined and synchronizedbiomechanical analysis and 3D tactical analysis of: (i) full game; (ii)strategy e.g. number of hits, type of stroke, location of hits and thelike; (iii) score; and (iv) match scoring patterns such as big points.The biomechanical analysis may be performed by combining (i) a type orstrength of stroke; (ii) type of motion or location of the tennis playerin the court; and (iii) type of behavioral pattern according to matchscoring patterns.

Additionally, embodiments of the present invention provide relativelyinexpensive system that may perform, without an operator, a preliminarysetup or corrections during the event, the following actions: (i)automatic calibration of cameras by identifying field boundaries; (ii)automatic customization of light in the stream of video, according to:(a) day and night lighting; and (b) weather conditions; (iii) automaticreduction of vibration in the stream of video; (iv) automaticidentification of start of the real sport game; (v) automaticidentification of end of set; and (vi) automatic identification of endof the real sport game. The system may utilize at least one computerdevice.

According to some embodiments of the present invention, there isprovided an automatic procedure of uploading of debriefing data to aninternet page of a player, and sharing performance of the player byproviding access to the debriefing data from any location via theInternet or any wireless network by laptops, tablets, smartphones etc.Further, remote debriefing may be available to players and coaches, thusoffering guidance in RT.

Furthermore, a dynamic updated database and a social network of athletesand sport players may create a platform for sport players to share datawith their coach, friends and family. The database may include forexample, clips of special events and statistics. Also, the database maybe used as a scouting tool for scouts, coaches and players.

According to some embodiments of the present invention, the smart-courtsystem may be implemented in multiple courts and all smart-court systemsmay be connected to one or more social networks and players' databases.

Moreover, embodiments of the present invention provide a system thatenables automatic profile generation according to performance of aTennis player during a game or during a series of games. The profile ofthe Tennis player that was generated may be used for: (i) scouting fornew talents; and (ii) finding a partner according to specifiedparameters. A social network of tennis players may be generated to allowsharing data with other players, a coach, friends and family via clips;special events and statistics e.g. speed of ball. Also, the system maybe used as a learning tool for players. The system may send automaticnotifications to scouts and to coaches, through the social network,according to update of players' performance parameters.

The generated profile of the real player may be published in the socialnetwork of tennis players. Talent scouting may be performed by utilizingthe published profile as well as partner matching. For example, a realplayer that is aware of her or his Tennis techniques that requireimprovement may search a partner that possesses a higher level in thesespecific Tennis techniques.

FIG. 1A is a top 3D perspective view diagram of a system that isincorporated in a Tennis court, according to some embodiments of theinvention. In a non-limiting example, the system may be incorporated ina Tennis court. A real player 110 may be training or competing against areal player 140 with a Tennis ball 120.

According to one aspect of the present invention, four cameras 130 maybe utilized to monitor a 3D area within and around a game's court tocapture motion and activities of real Tennis players 110 and 140 and ofthe ball 120. The cameras 130 may be megapixel cameras. When the systemis being activated, the video pixels that were captured are translatedby processor 150 into 3D coordinates for identifying the spatiallocation of the monitored objects. Based on said identified locations,the video data is processed for tracking the movement of the Tennis ball120 and the real players 110 and 140. The real players are not requiredto be equipped with an electronic sensing device. Since pixelcoordinates are known, automatic zoom on video is enabled.

According to some embodiments of the invention, camera calibration maybe enabled for each frame. The court may be divided into grids so onlyjunctions of the grids are being checked. If a deviation is identifiedin a grid than the frame may be checked for calibration. A slow framemay have a slow background and a fast frame may have a fast background.

According to another aspect of the invention, at least two cameras,preferably four cameras 130 may be automatically calibrated, accordingto identification of boundaries of the court. The automated calibrationmay not require an intervention of an operator. The cameras 130 maytrack the real players 110 and 140 and the Tennis ball 120 in anaccurate and reliable manner on any surface.

Furthermore, the stream of multimedia data (i.e. video) that istransmitted from the cameras 130, may be automatically customized andmay not require an intervention of an operator, according to: (a) dayand night lighting or artificial floodlight; and (b) weather conditions,for example, in windy weather the cameras 130 may wobble and themultimedia data that is transmitted might need to be fixed; and (c)shadow covering parts or most of the court. The system may also,automatically reduce vibration in the stream of video.

According to some embodiments of the invention, start and end of a realsport game or real training may be automatically identified by thesystem. Furthermore, the system may automatically identify change ofgame status, and may not require an intervention of an operator, byidentifying, for example, a predefined gesture of the real players 110and 140. The system may identify change of location of players 110 and140 when the real players 110 and 140 switch sides on the court. Also,the system may identify a repeated serve.

According to another aspect of the invention, the system may analyze apattern of behavior of real players 110 and 140 in a multimedia stream.The system may perform an analysis by receiving a stream of a video ofthe real sport game in RT. Next, the system may identify and tag theposition and the motion of real players 110 and 140, according toclassification rules. The classification rules may be based on tacticalbehavior pattern analysis of the position and motion of real players 110and 140.

Further, the system may enable a review of at least one of the objects'specified: (i) position; (ii) motion; and activity, according to theclassification. Furthermore, the system may store one or more tags whichare static information related to classified events the pattern oftactical behavior. Information regarding activity of the real playersmay include: (i) type of stroke; (ii) left or right hand usage; and(iii) strength of stroke.

According to yet another aspect of the invention, the system may providerecommendations to a player how to correct his or her stroke and/or gamemanagement and tactics thus, utilized as a learning tool of techniquesand strategies.

According to yet another aspect of the invention, the system'scapability to understand the course of the Tennis play based onidentifying and classifying events may contribute in cutting out idletime in the video file to save storage space. A Tennis match videoincludes long periods of idle time between games' session, where thereis not activity of the players which is relevant to the game. The systemmay identify the idle time and extract it from the video hence, reducingthe size of the video file, reducing storage, bandwidth and upload anddownload resources and cost.

According to yet another aspect of the invention, the system may play anabridged video or virtual 3D dynamic scenes representing the video gamesessions, in which the pattern of tactical behavior of the real players110 and 140 may be presented. The video and the related virtual 3Ddynamic scenes may be displayed simultaneously on the same screen,enabling the user to analyze the game both strategically by reviewingthe motion across the screen at the virtual 3D dynamic scene version,and analyzing biomechanical motion of part of the body at the videofile. The abridged video or virtual 3D dynamic scenes can be playedaccording to a user query for displaying only events of specific type,for example, showing all backend strokes through the game.

According to some embodiments of the invention, the user query may bemore specific by requesting all backend strokes at the front area of thecourt. Accordingly, a customized video (i.e. clip) may be createdincluding aggregation of events of specific type enabling the player andcoach to focus in specific type of events. The clip may be uploaded tothe social network and be accessed via the internet or it may be viewedand/or physically received on court.

According to yet another aspect of the invention, the customized videomay be generated by a specified criterion or event. For example,generating a video with all successful backhands in a real sport game orin a series of games. In another example, generating a video of allsuccessful forehands of all champion players from different geographicalplaces.

According to yet another aspect of the invention, the system may createan attribute profile of at least one of the real players 110 and 140based on at least one pattern of behavior and log of events. The patternof behavior may further be analyzed to identify a mental event. Anexample of a mental event that the system may recognize is when a realplayer is angry. Another mental event that the system may recognize maybe a decline in the performance of the players and conclude that it maybe due to a mental event. A delay in performance of a player orrepeatedly wiping sweat from a racket may also be interpreted as relatedto a mental event.

According to yet another aspect of the invention, the system is arrangedto be an integral part of a Tennis court. Further, the system may befully adaptable to both indoor and outdoor courts.

According to some embodiments of the present invention the real timeanalysis of events during the real sport game may be used for enouncingor displaying the game results for the players and the audience that iswatching the game via remote terminal 170. The real time analysis may betransferred via data communication link 136.

Optionally, the data related to the event may be used for supportingline calling decisions of a human umpire or as an objective officiatingtool. For example, the tool may provide accessible debriefing andanalysis capabilities combined with line-calling, to determine if a ballwas inside the court or not, in a Tennis game. The streamed data may be(i) location, speed of the tennis player or a ball and acceleration of aplayer or a ball; (ii) type or speed of a tennis stroke; (iii) result ofsuccess or failure at the game to the player; and (iv) data related toscore that were gained in the game in real time. Data about the scoringsystem is transmitted in real time. For example, type of points thatwere gained during the real sport game or training.

FIG. 1B is a block diagram of a smart-court system, according to someembodiments of the invention.

According to some embodiments of the invention, a smart-court system 190may be connected to cameras 130 via a data communication link 136. Thesmart-court system 190 may be comprised of an automatic recording system191 which automatically record and analyze a pattern of tacticalbehavior and then classify it, an event log file 194 and a dataprocessing system 192.

According to some embodiments of the invention, the data processingsystem 192 may be comprised of the following modules: (i) a capturingmodule 193 for grabbing the video of the RT sport session and encodingit; (ii) object detection module 200 (described in details in FIG. 2);(iii) a game tracker event 300 for identifying basic events; (iv) anevent module 400 for automatically identifying and classifying eventsbased on the tracking and analysis of the motion and the activities ofthe tracked objects and calculating statistics that occurred during theRT sport session; (v) a moments module 500 for creating customized video(see FIG. 5); (vi) an uploading module 600 for uploading profiles andvideo to the network; and (vii) debriefing presentation training module700 for enabling a user to watch the clip with the sequence of allevents from the same type or to watch the abridged video.

According to some embodiments of the invention, a server 155 may beconnected to the smart court system via data communication link 136. Theserver may be comprised of: (i) a profile database 157; (ii) socialmanagement module 158 for managing social activity between the players;and (iii) a remote coaching module 159 enabling coaching by view thegame video at remote location.

FIG. 2 is a flowchart diagram of object detection module 200 activity,according to some embodiments of the invention.

According to an aspect of the present invention, the object detectionmodule may begin with receiving a stream of multimedia data of a realsport game in Real Time (RT) (stage 210) from the capturing module 193.In each frame analysis the objects (i.e. players or ball) may beextracted from the background: first the foreground of each frame isidentified and extracted from the frame image. (stage 215). Optionally,the background of the image can be identified and extracted in some ofthe frames, as the background of the frame does not change from frame toframe it may be detected each specified amount of time.

At the next stage the objects are detected and extracted (the playersand the ball) in each frame (stage 220). Optionally, “noise” filter isapplied (stage 225) and identification of the objects video 3Dcoordinates is performed by analyzing stereo image of at least twocameras (stage 230).

Finally sending the identification and data of objects to a trackingmodule (stage 235).

FIG. 3 is a flowchart diagram illustrating system activity of gametracker module 300, according to some embodiments of the invention. Thegame tracker module 300 analyses frames to yield basic events such as astroke or a serve for the purpose of identifying a session. For example,identifying when a player strikes a ball before the ball touched theground. Additionally, the game tracker module 300 may also identify whenthe ball hit the net or when the ball fell out of the lines or when auser didn't respond to a ball that fell in the lines.

According to some embodiments of the invention, game tracker module 300may begin with receiving output results of each frame of player and ballcoordinates and identification of first stroke (serve) (stage 310). Thegame video recent frames are analyzed, during the RT sport game foridentifying basic events by identifying specific image pattern such asball touching ground stroke (stage 315) to determine start and end ofsession by identifying first stroke such a serve and identifying “out”,in, or miss event (stage 320).

Optionally, the travel route of the ball is detected and prediction ofits trajectory is calculated (stage 325).

At the end of each detected session, the analyzed basic event dataincluding coordinates of objects for each determined session of detectedbasic events is conveyed to an event module (stage 330).

FIG. 4 is a flowchart diagram illustrating activity of event module 400,according to some embodiments of the invention. The event module 400 mayyield complicated events out of analysis of several frames. For example,types of strokes and unforced error. First, the analyzed basic eventdata including coordinates of objects for each determined session isreceived from the tracking module 300.

According to some embodiments of the invention, event module 400 maybegin with using an event template which defines a set of predefinedparameters related to motion of objects (i.e. a player or a ball) (stage415).

At the next stage, the position motion and the activities of the trackedobjects that appear in the stream of multimedia data of each frame areanalyzed during the RT sport game (stage 420).

The received data parameters of each session received from the trackermodule and the motion and activity analysis are compared to eventtemplates parameters for identifying complex events in near realtime.(stage 425).

Based on the comparison results, the events are classified, wherein theclassification types are related to ruling of the game or related to thetype of stokes or to unforced error (stage 430).

For each event, the timing of beginning and end are estimated and asynchronized log of the event is generated (stages 435, 440), accordingto pattern definition e.g. identifying stroke touching point and ballcontact with the net or ground. Optionally, aggregating severalclassified events to analyze a pattern of behavior of a player toidentify event characteristics such as a mental event (stage 445) oraggregating several classified events to analyze a tactical behavior ofa player (stage 450).

Optionally, the event module 400 calculates and reports statistics ofevents and game related data throughout the game session (stage 455).

FIG. 5 is a flowchart diagram of Moments module 500, according to someembodiments of the invention.

According to some embodiments of the invention, a user may be enabled toconfigure preference of beginning and ending of an event. For example,one user may configure a beginning of backhand event when the player ismoving her or his hand and another user may configure beginning ofbackhand event when the ball leaves the racket of the player.

The moments modules receives the multimedia file of a real sport gameassociated with event log (stage 510). Optionally, the module enables todefine customized events segments based on user/system configuration andon determined events received information from event log includingclassification of the event (stage 515).

The module enables to create customized video movie by integrating eventor segments of events according to given criteria which may bedetermined by the user or predefined by the system, such criteria mayinclude selecting successful games session, fasted ball strikes, etc.(stage 520).

FIG. 6 is a flowchart diagram illustrating an uploading module 600,according to some embodiments of the invention.

According to an aspect of the present invention, after the real sportgame is over, the following stages are taken: Receiving video file of areal sport game associated with synchronized event log and customizedvideo (stage 610). Based on the game video, is created an attributeprofile of an object based on events log analysis and at least onepattern of behavior to be published in a social network (stage 615) andthe module enables a real-time upload of: the user profile, video fileof a real sport game and/or customized videos to a social network(stages 620, 625).

FIG. 7 is a flowchart diagram illustrating activity of debriefingpresentation training module 700, according to some embodiments of theinvention.

First the debriefing presentation training module receives a video fileof a real sport game associated with synchronized event log.

Throughout the game real time reports of events and statistics arereceived (stage 715) from the tracking module or the event module.

According to an aspect of the present invention, debriefing presentationtraining module 700 may receive a query from a user (720).

Optionally, the module plays a video file of a real sport game withsynchronized event log enabling the user to “jump” between specificevents according to user selections (stage 725).

Using the event log, an abridged video may be played per user query byjumping between requested events. The abridged video may present onlyscenarios including events related to the user query, such as, allstrokes at the rear part of the court (730). Throughout the presentationof the abridged video of the session the user may “jump” from one markedevent to the other based on the event log. According to user query basedon analyzed video data of objects motion, a virtual 3D dynamic scenesmovie can be played that is generated online or in real-time, using 3Dengine which simulates players and ball motion (step 735). Optionally,the module presents reports of events and statics in real time by voiceor by text appearing on the video (stage 740).

According to further embodiments of the present invention, the systemcomprises a pointing device (160, see FIG. 1) enabling to mark positionson the field ground for the purpose of indicating the position of thevirtual opponent. The pointing device is controlled by a designatedcontroller module, based on simulation of the opponent virtual playerand/or predefined training programs. According to an optionalembodiment, the pointing device can be a laser based device enabling touse a laser beam that continuously marks the location of the virtualopponent on the tennis court ground. According to another option thevirtual opponent is indicated by a hologram.

According to yet another aspect of the invention, the system may includeat least one tennis ball machine for projecting balls in a court fortraining purposes. The ball machine may be programmed to utilize thevideo game analysis for identifying player behavior and activate theball machine for effective training of the player.

According to some embodiments of the present invention, the patternanalysis of the players' image can indicate on physical characteristicsof the player such as the player's weight and calories that were lostthroughout the game.

According to an aspect of the invention, tracking, identifying andcapturing the motion of players and objects such as balls and racketsmay be performed utilizing the following elements: (i) RT methods; (ii)smart cameras; (iii) image processing and patterns recognition; (iv) 3Dobjects localization and tracking; (v) human behavior modeling; (vi) 3Dvisualization (vii) video compression; and (viii) web cloud platform.

Further, the tool may provide: (i) a RT ball speed detection; (ii) anaudio feedback, such as announcing game score or ball velocity; (iii)score measurement; (iv) in and out calculations; and (v) an immediatevideo playback.

In the above description, an embodiment is an example or implementationof the invention. The various appearances of “one embodiment”, “anembodiment” or “some embodiments” do not necessarily all refer to thesame embodiments.

Although various features of the invention may be described in thecontext of a single embodiment, the features may also be providedseparately or in any suitable combination. Conversely, although theinvention may be described herein in the context of separate embodimentsfor clarity, the invention may also be implemented in a singleembodiment.

Furthermore, it is to be understood that the invention can be carriedout or practiced in various ways and that the invention can beimplemented in embodiments other than the ones outlined in thedescription above.

The invention is not limited to those diagrams or to the correspondingdescriptions. For example, flow need not move through each illustratedbox or state, or in exactly the same order as illustrated and described.

Meanings of technical and scientific terms used herein are to becommonly understood as by one of ordinary skill in the art to which theinvention belongs, unless otherwise defined.

What is claimed is:
 1. A Smart-court system, comprising: a computerprogrammed to implement: a capture module, for receiving a video streamof a sport session; an object detector module, for tracking at least oneobject as appearing in the video stream; an event module, for analyzingmotion of the tracked at least one object, for identifying a pluralityof events and classifying the identified events into types based on saidtracking and analyzing, and for creating a synchronized event log of theclassified events based on said identifying and classifying; and apresentation module, for allowing a user to selectively play at leastone of the identified events classified into the types, by type, usingthe synchronized event log.
 2. The smart-court system of claim 1,further comprising a tracking module configured to identify a part ofthe sport session, the part starting with a ball stroke and ending witha player gaining a score.
 3. The smart-court system of claim 1, whereinone of the at least one tracked object is a player and the computer isfurther programmed to yield a pattern of behavior of the player frommore than one of the identified events.
 4. The smart-court system ofclaim 3, wherein the computer is further programmed to generate aprofile of the player, the profile comprising information based at leastin part, on the yielded pattern of behavior.
 5. The smart-court systemof claim 1, wherein said computer is further programmed to performautomatic calibration by dividing a court as appearing in the videostream using at least one grid, and in each frame of the video stream,checking the grid for deviation of a line of the court from the grid. 6.The smart-court system of claim 1, wherein the computer is furtherprogrammed to identify a start and end of a sport game or trainingsession and to automatically identify a change of status of the sportgame or training session.
 7. The smart-court system of claim 1, whereinthe computer is further programmed to identify a change of location ofplayers when the players switch sides on a court.
 8. The smart-courtsystem of claim 1, further comprising an additional techniques andstrategies module, configured to provide recommendations to a player onhow to correct activity in a sport game based on said analyzing.
 9. Thesmart-court system of claim 1, wherein said computer is furtherprogrammed to use said identifying of the events, for cutting out idletime in the video stream.
 10. The smart-court system of claim 1, whereinsaid presentation module is further configured to enable playing anabridged video displaying only some of the identified events by jumpingbetween some of the identified events according to a user-input queryand the synchronized log event.
 11. The smart-court system of claim 1,wherein the presentation module is further configured to play a virtual3D dynamic scene using a 3D engine which simulates objects behaviorbased on said analyzing.
 12. The smart-court system of claim 1, whereinsaid computer is further configured to identify occurrence of a mentalevent based on said analyzing.
 13. A method for implementing a smartcourt system, the method comprising: by a computer, receiving a videostream of a sport session; tracking at least one object as appearing inthe video stream; analyzing motion of the tracked at least one object;identifying a plurality of events and classifying the identified eventsinto types, based on said tracking and analyzing; creating asynchronized event log of the classified events based on saididentifying and classifying; and allowing a user to selectively play atleast one of the identified events classified into types, by type, usingthe synchronized event log.
 14. The method of claim 13, furthercomprising identifying a part of the sport session, the part startingwith a ball stroke and ending with a player gaining a score.
 15. Themethod of claim 13, further comprising yielding a pattern of behavior ofa player from more than one of the identified events.
 16. The method ofclaim 13, further comprising identifying a change of status of the sportsession according to a predefined gesture of a user.
 17. The method ofclaim 13, further comprising creating a customized video by integratingevents according to a given criteria defined by a user.
 18. The methodof claim 13, further comprising identifying unforced error based onidentifying timing and position of a player before striking a ball. 19.The method of claim 13, further comprising automatically calculating agame score based on said identified and classified events and on ananalysis of player and ball position and motion.
 20. The method of claim13, further comprising providing performance matching between players.